[ASM] Update ruleset to version 1.13.2 #6218

NachoEchevarria · 2024-10-29T15:09:16Z

Summary of changes

This PR updates the WAF's default ruleset to it's latest version: 1.13.2

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging · 2024-10-29T15:33:10Z

Datadog Report

Branch report: nacho/updateRuleset1.13.2
Commit report: c4edd27
Test service: dd-trace-dotnet

✅ 0 Failed, 363504 Passed, 2075 Skipped, 15h 5m 21.49s Total Time
⌛ 1 Performance Regression

⌛ Performance Regressions vs Default Branch (1)

WriteAndFlushEnrichedTraces - Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark 782.72µs (+95.98µs, +14%) - Details

andrewlock · 2024-10-29T15:41:33Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6218) - mean (71ms)  : 67, 74
     .   : milestone, 71,
    master - mean (70ms)  : 68, 73
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (6218) - mean (1,118ms)  : 1097, 1139
     .   : milestone, 1118,
    master - mean (1,116ms)  : 1096, 1136
     .   : milestone, 1116,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6218) - mean (110ms)  : 106, 114
     .   : milestone, 110,
    master - mean (109ms)  : 106, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (6218) - mean (777ms)  : 760, 795
     .   : milestone, 777,
    master - mean (771ms)  : 755, 787
     .   : milestone, 771,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6218) - mean (93ms)  : 90, 97
     .   : milestone, 93,
    master - mean (93ms)  : 90, 95
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (6218) - mean (734ms)  : 720, 747
     .   : milestone, 734,
    master - mean (731ms)  : 712, 750
     .   : milestone, 731,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6218) - mean (191ms)  : 187, 194
     .   : milestone, 191,
    master - mean (190ms)  : 187, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6218) - mean (1,203ms)  : 1179, 1227
     .   : milestone, 1203,
    master - mean (1,201ms)  : 1176, 1227
     .   : milestone, 1201,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6218) - mean (277ms)  : 272, 282
     .   : milestone, 277,
    master - mean (277ms)  : 272, 282
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6218) - mean (944ms)  : 924, 965
     .   : milestone, 944,
    master - mean (940ms)  : 926, 954
     .   : milestone, 940,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6218) - mean (265ms)  : 260, 270
     .   : milestone, 265,
    master - mean (265ms)  : 261, 268
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (6218) - mean (925ms)  : 907, 943
     .   : milestone, 925,
    master - mean (926ms)  : 906, 946
     .   : milestone, 926,

github-actions · 2024-10-29T16:14:55Z

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

1 occurrences of :

-        "_dd.appsec.event_rules.version": "1.13.1",
+        "_dd.appsec.event_rules.version": "1.13.2",

30 occurrences of :

-      _dd.appsec.event_rules.version: 1.13.1,
+      _dd.appsec.event_rules.version: 1.13.2,

andrewlock · 2024-10-29T17:45:33Z

Benchmarks Report for tracer 🐌

Benchmarks for #6218 compared to master:

2 benchmarks are faster, with geometric mean 1.208
1 benchmarks are slower, with geometric mean 1.143
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.67μs	40.2ns	201ns	0.0157	0.00785	0	5.43 KB
master	`StartStopWithChild`	netcoreapp3.1	9.89μs	47.2ns	189ns	0.0311	0.0155	0	5.61 KB
master	`StartStopWithChild`	net472	16.2μs	47.4ns	184ns	1.02	0.307	0.0944	6.07 KB
#6218	`StartStopWithChild`	net6.0	7.75μs	43.2ns	317ns	0.0149	0.00373	0	5.43 KB
#6218	`StartStopWithChild`	netcoreapp3.1	9.88μs	54.9ns	347ns	0.0151	0.00502	0	5.62 KB
#6218	`StartStopWithChild`	net472	16.5μs	54.5ns	211ns	1.03	0.318	0.0978	6.06 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	488μs	454ns	1.7μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	633μs	281ns	1.01μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	838μs	396ns	1.48μs	0.417	3.3 KB
#6218	`WriteAndFlushEnrichedTraces`	net6.0	471μs	430ns	1.66μs	0	2.7 KB
#6218	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	653μs	290ns	1.12μs	0	2.7 KB
#6218	`WriteAndFlushEnrichedTraces`	net472	841μs	604ns	2.26μs	0.422	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	203μs	1.11μs	6.57μs	0.196	18.45 KB
master	`SendRequest`	netcoreapp3.1	225μs	1.29μs	11.6μs	0.229	20.61 KB
master	`SendRequest`	net472	0.000902ns	0.000448ns	0.00174ns	0	0 b
#6218	`SendRequest`	net6.0	208μs	1.22μs	11.8μs	0.2	18.45 KB
#6218	`SendRequest`	netcoreapp3.1	241μs	1.41μs	13.3μs	0.219	20.61 KB
#6218	`SendRequest`	net472	0.000946ns	0.000429ns	0.0016ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6218

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	1.143	677,992.11	774,870.15

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	565μs	2.63μs	9.85μs	0.59	0	0	41.62 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	676μs	2.46μs	8.85μs	0.342	0	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	net472	858μs	3.12μs	12.1μs	8.19	2.59	0.431	53.28 KB
#6218	`WriteAndFlushEnrichedTraces`	net6.0	580μs	3.22μs	20.4μs	0.566	0	0	41.53 KB
#6218	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	783μs	4.55μs	38.9μs	0.393	0	0	41.8 KB
#6218	`WriteAndFlushEnrichedTraces`	net472	879μs	4.12μs	16μs	8.25	2.6	0.434	53.34 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.2μs	1.62ns	6.29ns	0.0143	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	1.44ns	5.18ns	0.0133	1.02 KB
master	`ExecuteNonQuery`	net472	2.08μs	1.77ns	6.84ns	0.157	987 B
#6218	`ExecuteNonQuery`	net6.0	1.32μs	1.09ns	4.22ns	0.0139	1.02 KB
#6218	`ExecuteNonQuery`	netcoreapp3.1	1.78μs	1.32ns	5.1ns	0.0133	1.02 KB
#6218	`ExecuteNonQuery`	net472	2.06μs	1.64ns	6.35ns	0.156	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6218

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑net6.0	1.171	1,382.79	1,181.14

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.29μs	0.612ns	2.37ns	0.0136	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.51μs	0.658ns	2.46ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.64μs	1.37ns	5.29ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.38μs	0.946ns	3.54ns	0.0131	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.66μs	1.66ns	6.21ns	0.014	1.02 KB
master	`CallElasticsearchAsync`	net472	2.64μs	1.13ns	4.36ns	0.167	1.05 KB
#6218	`CallElasticsearch`	net6.0	1.21μs	1.9ns	7.35ns	0.0138	976 B
#6218	`CallElasticsearch`	netcoreapp3.1	1.52μs	1.32ns	4.76ns	0.013	976 B
#6218	`CallElasticsearch`	net472	2.46μs	2.71ns	10.5ns	0.158	995 B
#6218	`CallElasticsearchAsync`	net6.0	1.18μs	1.29ns	4.66ns	0.0136	952 B
#6218	`CallElasticsearchAsync`	netcoreapp3.1	1.72μs	1.15ns	4.45ns	0.0137	1.02 KB
#6218	`CallElasticsearchAsync`	net472	2.54μs	1.69ns	6.53ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.2μs	0.639ns	2.48ns	0.0132	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.62μs	0.81ns	3.03ns	0.0129	952 B
master	`ExecuteAsync`	net472	1.75μs	0.507ns	1.97ns	0.145	915 B
#6218	`ExecuteAsync`	net6.0	1.32μs	0.675ns	2.52ns	0.0133	952 B
#6218	`ExecuteAsync`	netcoreapp3.1	1.57μs	0.716ns	2.68ns	0.0123	952 B
#6218	`ExecuteAsync`	net472	1.8μs	1.29ns	4.99ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.22μs	1.68ns	6.27ns	0.0316	2.22 KB
master	`SendAsync`	netcoreapp3.1	5.13μs	2.45ns	8.82ns	0.0359	2.76 KB
master	`SendAsync`	net472	7.83μs	2.24ns	8.69ns	0.496	3.15 KB
#6218	`SendAsync`	net6.0	4.05μs	1.55ns	5.79ns	0.0301	2.22 KB
#6218	`SendAsync`	netcoreapp3.1	5.07μs	0.932ns	3.49ns	0.0355	2.76 KB
#6218	`SendAsync`	net472	7.72μs	2.66ns	10.3ns	0.499	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.53μs	0.835ns	2.89ns	0.0229	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.34μs	3.75ns	14ns	0.0218	1.64 KB
master	`EnrichedLog`	net472	2.67μs	0.889ns	3.33ns	0.249	1.57 KB
#6218	`EnrichedLog`	net6.0	1.46μs	0.846ns	3.17ns	0.0228	1.64 KB
#6218	`EnrichedLog`	netcoreapp3.1	2.2μs	4.42ns	17.1ns	0.0219	1.64 KB
#6218	`EnrichedLog`	net472	2.61μs	1.71ns	5.92ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	117μs	259ns	1μs	0.0585	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	185ns	715ns	0	0	4.28 KB
master	`EnrichedLog`	net472	152μs	338ns	1.31μs	0.678	0.226	4.46 KB
#6218	`EnrichedLog`	net6.0	119μs	159ns	615ns	0.0596	0	4.28 KB
#6218	`EnrichedLog`	netcoreapp3.1	122μs	342ns	1.32μs	0	0	4.28 KB
#6218	`EnrichedLog`	net472	153μs	318ns	1.23μs	0.689	0.23	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3μs	0.782ns	3.03ns	0.03	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.21μs	2.31ns	8.95ns	0.0294	2.2 KB
master	`EnrichedLog`	net472	4.85μs	1.18ns	4.56ns	0.319	2.02 KB
#6218	`EnrichedLog`	net6.0	3.13μs	0.562ns	2.18ns	0.0298	2.2 KB
#6218	`EnrichedLog`	netcoreapp3.1	4.23μs	1.85ns	7.17ns	0.0297	2.2 KB
#6218	`EnrichedLog`	net472	4.95μs	1.13ns	4.22ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.36μs	1.21ns	4.68ns	0.0163	0	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.71μs	0.949ns	3.68ns	0.0155	0	1.14 KB
master	`SendReceive`	net472	2.08μs	1.13ns	4.24ns	0.184	0	1.16 KB
#6218	`SendReceive`	net6.0	1.42μs	0.482ns	1.8ns	0.0158	0	1.14 KB
#6218	`SendReceive`	netcoreapp3.1	1.78μs	1.04ns	4.04ns	0.0149	0	1.14 KB
#6218	`SendReceive`	net472	2.15μs	1.21ns	4.68ns	0.183	0.00108	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.68μs	0.749ns	2.9ns	0.0215	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.86μs	1.5ns	5.79ns	0.0212	1.65 KB
master	`EnrichedLog`	net472	4.3μs	3.23ns	12.5ns	0.322	2.04 KB
#6218	`EnrichedLog`	net6.0	2.73μs	2.01ns	7.52ns	0.0217	1.6 KB
#6218	`EnrichedLog`	netcoreapp3.1	3.84μs	2.54ns	9.15ns	0.0229	1.65 KB
#6218	`EnrichedLog`	net472	4.35μs	4.36ns	16.9ns	0.324	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6218

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.246	804.72	645.95

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	417ns	0.222ns	0.859ns	0.00804	576 B
master	`StartFinishSpan`	netcoreapp3.1	654ns	1.26ns	4.89ns	0.00783	576 B
master	`StartFinishSpan`	net472	675ns	0.735ns	2.75ns	0.0917	578 B
master	`StartFinishScope`	net6.0	483ns	0.469ns	1.82ns	0.00971	696 B
master	`StartFinishScope`	netcoreapp3.1	805ns	0.344ns	1.33ns	0.00926	696 B
master	`StartFinishScope`	net472	933ns	1.02ns	3.94ns	0.105	658 B
#6218	`StartFinishSpan`	net6.0	410ns	0.18ns	0.698ns	0.00803	576 B
#6218	`StartFinishSpan`	netcoreapp3.1	596ns	0.424ns	1.64ns	0.00781	576 B
#6218	`StartFinishSpan`	net472	671ns	0.343ns	1.33ns	0.0916	578 B
#6218	`StartFinishScope`	net6.0	505ns	0.2ns	0.722ns	0.00972	696 B
#6218	`StartFinishScope`	netcoreapp3.1	646ns	0.599ns	2.32ns	0.0094	696 B
#6218	`StartFinishScope`	net472	861ns	0.291ns	1.13ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	625ns	0.693ns	2.68ns	0.00977	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	910ns	2.06ns	7.97ns	0.00893	696 B
master	`RunOnMethodBegin`	net472	1.09μs	0.792ns	3.07ns	0.104	658 B
#6218	`RunOnMethodBegin`	net6.0	649ns	0.356ns	1.38ns	0.0097	696 B
#6218	`RunOnMethodBegin`	netcoreapp3.1	978ns	0.472ns	1.77ns	0.00916	696 B
#6218	`RunOnMethodBegin`	net472	1.09μs	1.34ns	5.17ns	0.104	658 B

update ruleset

de0409d

NachoEchevarria added the area:asm label Oct 29, 2024

update snapshots

c4edd27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ASM] Update ruleset to version 1.13.2 #6218

[ASM] Update ruleset to version 1.13.2 #6218

NachoEchevarria commented Oct 29, 2024 •

edited

Loading

datadog-ddstaging bot commented Oct 29, 2024 •

edited

Loading

andrewlock commented Oct 29, 2024 •

edited

Loading

github-actions bot commented Oct 29, 2024

andrewlock commented Oct 29, 2024

Raw results

Raw results

Raw results

Slower ⚠️ in #6218

Raw results

Raw results

Faster 🎉 in #6218

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6218

Raw results

Raw results

[ASM] Update ruleset to version 1.13.2 #6218

Are you sure you want to change the base?

[ASM] Update ruleset to version 1.13.2 #6218

Conversation

NachoEchevarria commented Oct 29, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Oct 29, 2024 • edited Loading

Datadog Report

⌛ Performance Regressions vs Default Branch (1)

andrewlock commented Oct 29, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

github-actions bot commented Oct 29, 2024

Snapshots difference summary

andrewlock commented Oct 29, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Slower ⚠️ in #6218

Raw results

Raw results

Faster 🎉 in #6218

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6218

Raw results

Raw results

NachoEchevarria commented Oct 29, 2024 •

edited

Loading

datadog-ddstaging bot commented Oct 29, 2024 •

edited

Loading

andrewlock commented Oct 29, 2024 •

edited

Loading